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DEFENSE  TRANSPORTATION 

Study  Limitations  Raise  Questions  about 
the  Adequacy  and  Completeness  of  the 
Mobility  Capabilities  Study  and  Report 


Why  GAO  Did  This  Study 

The  Department  of  Defense  (DOD) 
issued  the  Mobility  Capabilities 
Study  (MCS),  which  was  intended 
to  identify  and  quantify  the  mobility 
capabilities  needed  to  support  U.S. 
strategic  objectives  into  the  next 
decade.  The  MCS  found  that 
projected  capabilities  are  adequate 
to  achieve  U.S.  objectives  with  an 
acceptable  level  of  risk — that  is, 
current  U.S.  inventory  of  aircraft, 
ships,  prepositioned  assets,  and 
other  capabilities  are  sufficient,  in 
conjunction  with  host  nation 
support,  and  assuming  planned 
investments  take  place. 

The  Senate  report  accompanying 
the  bill  for  the  fiscal  year  2005 
Defense  Authorization  Act  required 
GAO  to  report  on  the  adequacy  and 
completeness  of  the  MCS.  GAO 
assessed  the  extent  to  which  the 
MCS  met  generally  accepted 
research  standards  that  this  type  of 
study  would  be  expected  to  meet  to 
be  considered  sound  and  complete. 


What  GAO  Recommends 


GAO  recommends  that  the 
Secretary  of  Defense,  in  future 
mobility  s  capabilities  studies 
beginning  with  any  study  currently 
under  way,  develop  models  and 
data  for  critical  missions  and 
processes;  include  in  study  reports 
an  explanation  of  how  stated 
limitations  might  impact  results; 
and,  incorporate  both  mobility  and 
warfighting  metrics  to  determine 
capabilities.  In  comments,  DOD 
concurred  with  two  of  the 
recommendations  and  claimed  they 
did  not  understand  the  third,  which 
GAO  clarified. 

www.gao.gov/cgi-bin/getrpt7GAO-06-938. 

To  view  the  full  product,  including  the  scope 
and  methodology,  click  on  the  link  above. 

For  more  information,  contact  William  M.  Solis 
at  (202)  512-8365  orsolisw@gao.gov. 


What  GAO  Found 

DOD  used  an  innovative  approach  in  conducting  the  study  and 
acknowledged  methodological  limitations  in  its  report;  however,  it  did  not 
fully  disclose  how  these  limitations  could  affect  the  MCS  conclusions  and 
recommendations.  Therefore,  it  is  not  transparent  how  the  analyses  done  for 
the  study  support  DOD’s  conclusions.  Measured  against  relevant  generally 
accepted  research  standards,  GAO  has  identified  limitations  in  the  MCS  and 
its  report  that  raise  questions  about  their  adequacy  and  completeness.  GAO 
suggests  that  Congress  and  other  decision  makers  exercise  caution  in  using 
the  MCS  to  make  investment  decisions.  Among  GAO’s  findings: 

•  Aspects  of  modeling  and  data  were  inadequate  in  some  areas  because 
data  were  lacking  and  the  models  used  could  not  simulate  all  relevant 
aspects  of  the  missions.  The  report  did  not  explain  how  these  limitations 
could  affect  the  study  results  or  what  the  impact  on  projected  mobility 
capabilities  might  be.  Generally  accepted  research  standards  require  that 
models  used  are  adequate  for  the  intended  purpose,  represent  a 
complete  range  of  conditions,  and  that  data  used  are  properly  generated 
and  complete.  For  example,  the  MCS  modeled  hypothetical  homeland 
defense  missions  rather  than  homeland  defense  demands  derived  from  a 
well  defined  and  approved  concept  of  operations  for  homeland  defense, 
because  the  specific  details  of  the  missions  were  still  being  determined 
and  the  data  used  may  be  incomplete.  The  MCS  also  was  unable  to 
model  the  flexible  deterrent  options/deployment  order  process  to  move 
units  and  equipment  into  theater  because  of  lack  of  data,  but  the  study 
assumed  a  robust  use  of  this  process.  In  addition,  the  MCS  report 
contains  over  80  references  to  the  need  for  improved  modeling  or  data. 

•  While  the  MCS  concluded  that  combined  U.S.  and  host  nation 
transportation  assets  were  adequate,  in  describing  the  use  of  warfighting 
metrics  in  its  analyses,  the  report  does  not  provide  a  clear  understanding 
of  the  direct  relationship  of  warfighting  objectives  to  transportation 
capabilities.  Additionally,  the  report  stated  that  further  analysis  is 
required  to  understand  the  operational  impact  of  increased  or  decreased 
strategic  lift  on  achieving  warfighting  objectives.  Relevant  generally 
accepted  research  standards  require  that  conclusions  be  supported  by 
analyses.  The  use  of  both  warfighting  and  mobility  metrics  would  allow 
decision  makers  to  know  whether  combat  tasks  were  achieved  and  how 
much  strategic  transportation  is  needed  to  accomplish  those  tasks. 

•  In  some  cases,  the  MCS  results  were  incomplete,  unclear,  or  contingent 
on  further  study,  making  it  difficult  to  identify  findings  and  evaluate 
evidence.  Relevant  research  standards  require  results  to  be  presented  in 
a  complete,  accurate,  and  relevant  manner.  For  example,  the  report 
contains  recommendations  for  further  studies  and  assessments,  five  of 
which  are  under  way.  However,  DOD  has  no  plans  to  report  the  impact 
of  these  studies  on  the  MCS  results  after  the  studies  are  complete.  In 
addition,  the  report  contains  qualified  information  that  is  not  presented 
clearly,  such  as  varying  assessments  of  intra-theater  assets  in  three 
different  places. 
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United  States  Government  Accountability  Office 
Washington,  DC  20548 


September  20,  2006 
Congressional  Committees 

The  National  Security  Strategy  of  the  United  States  requires  global 
mobility  through  rapid,  effective,  and  efficient  projections  of  power  at 
home  and  abroad  to  deploy  and  sustain  America’s  armed  forces.  To 
improve  its  mobility  capabilities,  the  Department  of  Defense  (DOD)  plans 
to  spend  more  than  $60  billion  from  fiscal  years  2006  through  2011  for 
aircraft,  ships,  ground  transportation,  prepositioned  assets,  and  other 
mobility  assets.  DOD  has  conducted  several  studies  to  determine  mobility 
requirements  and  recently  completed  a  study  of  its  mobility  capabilities 
and  issued  a  report  in  December  2006.  The  intent  of  the  Mobility 
Capabilities  Study  (MCS)  was  to  identify  and  quantify  the  mobility 
capabilities  needed  to  support  U.S.  strategic  objectives  into  the  next 
decade.  The  MCS  determined  that  the  projected  mobility  capabilities  are 
adequate  to  achieve  U.S.  objectives  with  an  acceptable  level  of  risk  during 
the  period  from  fiscal  years  2007  through  2013;  that  is,  the  current  U.S. 
inventory  of  aircraft,  ships,  prepositioned  assets,  and  other  capabilities  are 
sufficient,  in  conjunction  with  host  nation  support.  The  MCS  emphasized 
that  continued  investment  in  the  mobility  system,  in  line  with  current 
departmental  priorities  and  planned  spending,  is  required  to  maintain 
these  capabilities  in  the  future.  This  includes,  for  example,  fully  funding 
Army  prepositioned  assets  as  planned  and  completing  a  planned 
reengineering  of  the  C-5  aircraft.  The  MCS  report  also  made 
recommendations  to  conduct  further  studies,  develop  plans  and  strategies, 
and  improve  data  collection  and  mobility  models.  In  fact,  DOD  officials 
told  us  that  a  Mobility  Capabilities  Study-2006  is  underway. 

In  the  Senate  report  accompanying  the  bill  for  the  fiscal  year  2006  Defense 
Authorization  Act,  you  asked  us  to  monitor  the  process  used  to  conduct 
the  MCS  and  report  on  the  adequacy  and  completeness  of  the  study.1 
Specifically,  our  objective  was  to  determine  whether  the  MCS  was 
adequate  and  complete.  On  March  1,  2006,  we  briefed  your  staff  on  our 
preliminary  observations.  This  report  expands  on  that  briefing  and  makes 
recommendations  to  the  Secretary  of  Defense. 


1  S.  Rep.  108-260,  at  126  (2004). 
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To  conduct  our  review  of  the  MCS,  we  analyzed  the  final  MCS  report,  the 
MCS  Terms  of  Reference  and  MCS  Study  Plan,  as  well  as  other  DOD 
policies  and  guidance  concerning  how  DOD  would  conduct  the  MCS  and 
the  databases  and  models  used  in  the  study.  We  identified  generally 
accepted  research  standards  that  define  a  sound  and  complete  quality 
study  that  were  relevant  to  the  MCS,  and  assessed  the  extent  to  which  the 
MCS  report  met  these  standards.  We  interviewed  study  officials,  study 
participants,  and  subject  matter  experts  from  several  DOD  entities, 
including  the  combatant  commands  and  the  military  services.  As  we 
monitored  the  development  of  the  MCS,  we  requested  that  DOD  provide 
documentation  supporting  and  verifying  key  analytical  and  decision¬ 
making  processes.2  DOD  officials  could  not  produce  this  documentation 
during  the  development  of  the  MCS  or  following  issuance  of  the  report. 
Consequently,  we  were  unable  to  fully  determine  whether  the  analytical 
and  decision-making  processes  that  we  believe  are  significant  to  the 
credibility  of  the  study  supported  the  MCS  effort  and  its  conclusions.  Our 
scope  and  methodology  are  discussed  in  more  detail  in  appendix  I.  We 
conducted  our  work  from  July  2004  through  July  2006  in  accordance  with 
generally  accepted  government  auditing  standards. 


Results  in  Brief 


DOD  used  an  innovative  approach  in  conducting  the  study  and 
acknowledged  some  methodological  limitations  in  its  report,  as  any  sound 
study  should.  However,  it  did  not  fully  disclose  how  these  limitations 
could  affect  the  MCS  conclusions  and  recommendations.  Therefore,  it  is 
not  transparent  how  the  analyses  done  for  the  study  support  DOD’s 
conclusions.  As  measured  against  relevant  generally  accepted  research 
standards,  we  identified  limitations  in  the  MCS  study  and  report  that  raise 
questions  about  their  adequacy  and  completeness.  Among  our  findings: 


"  Examples  of  the  documentation  we  requested  to  support  and  verify  key  analytical  and 
decision-making  processes  used  by  DOD  to  conduct  the  MCS  included  (1)  the  accreditation 
report  and  supporting  documentation  or  evidence  of  the  verification,  validation,  and 
accreditation  process  for  the  models  and  data  used  in  the  MCS;  (2)  copies  of  MCS  working 
group  meeting  minutes  that  verify  and  validate  the  analytical  processes  the  various  MCS 
study  teams  and  study  participants  used  to  vet  and  agree  upon  data,  scenarios, 
assumptions,  models,  and  associated  risk;  and  (3)  copies  of  MCS  General  Officer  Steering 
Committee  and  Executive  Committee  meeting  minutes  that  verify  and  validate  the 
analytical  and  decision-making  processes  the  DOD  senior  leadership  used  to  vet  and  agree 
upon  the  key  data,  scenarios,  assumptions,  models,  and  associated  risk  used  to  conduct  the 
MCS,  as  well  as  agreement  with  the  study  results. 
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•  Aspects  of  modeling  and  data  were  inadequate  in  some  areas  because  data 
were  lacking  and  some  of  the  models  used  could  not  simulate  all  relevant 
aspects  of  the  missions.  The  report  did  not  explain  how  these  limitations 
could  affect  the  study  results  or  what  the  effect  on  the  projected  mobility 
capabilities  might  be.  Relevant  research  standards  require  that  models 
used  are  adequate  for  the  intended  purpose,  represent  a  complete  range  of 
conditions,  and  that  data  used  are  properly  generated  and  complete.  For 
example,  the  MCS  modeled  hypothetical  homeland  defense  missions 
rather  than  homeland  defense  demands  derived  from  a  well  defined  and 
approved  concept  of  operations  for  homeland  defense,  because  the 
specific  details  of  the  missions  were  still  being  determined,  and  DOD 
acknowledged  that  the  data  used  may  be  incomplete.  The  MCS  also  was 
unable  to  model  the  flexible  deterrent  options/deployment  order  process 
to  move  units  and  equipment  into  theater  due  to  lack  of  data,  but  the  study 
assumed  a  robust  use  of  this  process,  which  in  one  scenario  accounted  for 
approximately  60  percent  of  the  airlift  prior  to  beginning  combat 
operations.3  In  addition,  the  MCS  report  contains  more  than  80  references 
to  the  need  for  improved  modeling,  and  12  of  these  references  call  for 
additional  data  or  other  refinements.  Additionally,  the  MCS  modeled  the 
year  2012  to  determine  the  transportation  capabilities  needed  for  the  years 
2007  through  2013.  The  year  2012  did  not  place  as  much  demand  for 
mobility  assets  in  support  of  smaller  military  operations,  such  as 
peacekeeping,  as  other  years.  However,  DOD  officials  considered  2012 — 
the  year  modeled — as  “most  likely”  to  occur  and  stated  that  statistically  it 
was  not  different  from  other  years  in  the  2007  to  2013  period  even  though 
the  number  of  smaller  military  operations  is  the  least  of  any  of  the  years 
reviewed. 

•  While  the  MCS  concluded  that  combined  U.S.  and  host  nation 
transportation  assets  were  adequate  to  meet  U.S.  objectives  with 
acceptable  risk,  the  report,  in  describing  the  use  of  warfighting  metrics  in 
its  analyses,  does  not  provide  a  clear  understanding  of  the  direct 
relationship  of  warfighting  objectives  to  transportation  capabilities. 
Acknowledging  this  point,  the  report  stated  that  further  analysis  is 
required  to  understand  the  operational  impact  of  increased  or  decreased 
strategic  lift  on  achieving  warfighting  objectives.  Relevant  generally 
accepted  research  standards  require  that  conclusions  be  supported  by 
analyses.  The  use  of  warfighting  metrics  is  a  measure  to  determine 
whether  combat  tasks,  such  as  achieving  air  superiority,  are  achieved. 


3  Deployment  orders  are  issued  to  deploy  specific  capabilities  as  commitment  decisions  are 
made,  rather  than  a  deploying  unit’s  full  set  of  equipment  or  capabilities.  Flexible  Deterrent 
Options  (FDOs)  provide  escalation  options  during  the  initial  stages  of  a  conflict.  FDOs  are 
employed  under  certain  conditions  to  deter  adversarial  actions  contrary  to  U.S.  interests. 
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However,  they  do  not  measure  whether  appropriate  personnel,  supplies, 
and  equipment  arrived  in  accordance  with  timelines.  As  a  result,  we  could 
not  determine  how  the  study  concluded  that  planned  transportation  assets 
were  adequate  because  the  study  did  not  contain  a  transparent  analysis  to 
support  its  conclusion  or  a  clear  roadmap  in  the  report  to  help  decision 
makers  understand  what  that  conclusion  meant  in  terms  of  type  and 
number  of  mobility  assets  needed.  Previous  DOD  mobility  studies 
primarily  used  mobility  metrics,  which  measured  success  in  terms  of  tons 
of  equipment  and  personnel  moved  per  day  to  accomplish  military 
objectives.  The  use  of  both  warfighting  and  mobility  metrics  to  measure 
success  would  allow  decision  makers  to  know  whether  combat  tasks  were 
achieved  and  how  much  strategic  transportation  is  needed  to  accomplish 
those  tasks. 

•  In  some  cases,  the  MCS  results  were  incomplete,  unclear,  or  contingent  on 
further  study,  making  it  difficult  to  identify  findings  and  evaluate  evidence. 
Relevant  research  standards  require  results  to  be  presented  in  a  complete, 
accurate,  and  relevant  manner.  For  example,  the  report  contains  several 
recommendations  for  further  studies  and  assessments,  five  of  which  are 
under  way.  However,  DOD  has  no  plans  to  report  the  effect  of  these 
studies  on  the  MCS  results  after  the  studies  are  complete.  In  addition,  the 
report  contains  qualified  information  that  is  not  presented  clearly,  such  as 
varying  assessments  of  intratheater  assets  in  three  different  places  in  the 
report.  The  lack  of  clarity  and  conciseness  of  the  reported  results  can  limit 
the  study’s  usefulness  to  decision  makers  and  stakeholders. 

•  Verification,  validation,  and  accreditation  (W&A)  of  models  and  data  used 
to  conduct  the  study  was  not  complete  because  it  was  not  done  in 
accordance  with  DOD  policy  or  relevant  research  standards.  Moreover, 
relevant  research  standards  state  that  a  study  report  should  include  a 
W&A  accreditation  report  that  is  signed  by  the  study  director  and 
addresses  the  models  and  data  certification.  DOD  officials  acknowledged 
that  they  did  not  comply  with  DOD  W&A  policy  when  using  legacy 
models  in  the  MCS  because  they  contended  that  long-term  use  of  models 
and  data  constitutes  an  equivalent  W&A  process.  Other  than  a  description 
of  the  process  contained  in  the  MCS  report,  DOD  officials  could  provide 
no  additional  documentation  to  verify  and  validate  this  equivalent  process 
to  provide  the  assurance  that  models  and  data  used  in  the  MCS  reduced 
the  risk  inherent  in  modeling  and  simulation  and  added  to  the  credibility 
of  the  results.  Moreover,  officials  could  not  provide  documentation  to 
support  key  analytical  and  decision-making  processes  used  by  senior  DOD 
leadership,  thus  undermining  the  credibility  of  the  reported  study  results. 

These  limitations  to  the  study’s  methodology  raise  questions  concerning 
the  accuracy  of  the  study’s  finding  that  projected  capabilities  are  adequate 
to  achieve  U.S.  objectives  with  an  acceptable  level  of  risk.  Until  DOD 


Page  4 


GAO-06-938  Defense  Transportation 


conducts  an  adequate  and  complete  future  MCS  and  clearly  discloses  all 
limitations  and  their  effects  on  the  study  results,  decision  makers  may  be 
unable  to  clearly  understand  the  operational  implications  of  the  study 
results  and  make  fully  informed  programmatic  investment  decisions 
concerning  mobility  capabilities.  We  are  recommending  that  the  Secretary 
of  Defense,  when  conducting  future  mobility  capabilities  studies, 
beginning  with  any  study  currently  under  way,  develop  and  use  models 
and  data  for  critical  missions  and  processes  that  are  verified,  validated, 
and  accredited  as  required;  include  in  study  reports  an  explanation  of  how 
stated  limitations  might  impact  the  study  results  and,  at  a  minimum, 
describe  how  recommended  future  studies  might  be  conducted  to  enhance 
the  results  of  the  original  study;  and  incorporate  both  mobility  and 
warfighting  metrics  in  determining  capabilities. 

In  commenting  on  a  draft  of  this  report,  DOD  concurred  with  the  first  and 
third  recommendations  and  claimed  that  they  did  not  understand  the 
second.  We  have  clarified  that  recommendation  to  the  Secretary  of 
Defense  to  include  in  study  reports  an  explanation  of  how  stated 
limitations  might  impact  the  study  results  and,  at  a  minimum,  describe 
how  recommended  future  studies  might  be  conducted  to  enhance  the 
results  of  the  original  study.  In  its  comments,  DOD  also  stated  that  the 
report  contained  misleading  information  and  factual  errors.  We  disagree 
with  DOD’s  assertion.  We  did  modify  our  report  to  respond  to  a  DOD 
technical  comment  related  to  homeland  defense  missions.  DOD’s 
comments  and  our  evaluation  of  them  are  discussed  in  the  agency 
comments  section  of  this  report. 


Background 


The  MCS  was  the  first  assessment  of  DOD’s  mobility  system  since  2000. 
The  study  was  designed  to  identify  changes  in  DOD’s  transportation  force 
structure  due  to  changes  in  threats  and  national  security  and  military 
strategies.  The  MCS  is  the  fourth  in  a  series  of  major  mobility  studies  that 
DOD  has  conducted  since  the  end  of  the  Cold  War.  The  first  study,  the 
Mobility  Requirements  Study,  conducted  in  1992,  was  undertaken  because 
of  concern  about  the  DOD’s  strategic  mobility  capabilities  in  the  wake  of 
Operation  Desert  Shield  and  Operation  Desert  Storm.  That  study 
established  mobility  requirements  for  the  post-Cold  War  era;  defined 
baseline  requirements  for  intertheater,  or  strategic,  mobility;  and  proposed 
a  long-range  investment  plan  to  meet  these  requirements.  The  Mobility 
Requirements  Study  Bottom-Up  Review  Update,  conducted  in  1994, 
reaffirmed  the  need  for  increases  in  key  mobility  components  and 
validated  the  prior  study’s  recommendation  for  the  procurement  of 
additional  ships  for  afloat  prepositioning  and  for  surge  deployments  of 
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forces  based  in  the  continental  United  States.  The  Mobility  Requirements 
Study — 2005,  issued  in  2001,  projected  future  mobility  requirements  based 
on  two  nearly  simultaneous  major  regional  contingencies.  It  included  a 
broader  range  of  factors,  including  host  nation  support  and  enemy  use  of 
weapons  of  mass  destruction,  than  the  previous  studies. 

The  current  MCS,  which  began  in  May  2004,  reassessed  DOD’s  mobility 
capabilities  against  the  backdrop  of  a  revised  National  Military  Strategy 
that  included  the  ongoing  war  against  violent  extremism,  an  evolving 
global  defense  posture,  a  new  force-sizing  construct,  revised  campaign 
scenarios,  and  ongoing  departmentwide  transformation  efforts.  The  study 
results  were  intended  to  support  decisions  on  future  strategic  airlift,  aerial 
refueling  aircraft,  and  sealift  procurements  needed  to  meet  varying 
military  requirements.  The  study  used  an  innovative  “capabilities-based” 
approach,  measuring  existing  and  currently  projected  mobility  capabilities 
against  warfighting  demands  that  could  be  expected  in  fiscal  year  2012 
while  also  considering  mobility  demands  during  the  7-year  period  from 
fiscal  year  2007  through  fiscal  year  2013.  According  to  DOD  officials,  the 
Secretary  of  Defense  believed  this  approach  would  give  him  greater 
flexibility  in  deciding  which  capabilities  to  fund  in  a  constrained  budget 
environment.  In  considering  each  aspect  of  the  National  Military  Strategy,4 
the  MCS  modeled  warfighting  scenarios  in  the  year  2012  using  different 
premises  with  varying  assumptions  to  develop  and  evaluate  mobility 
capability  mix  alternatives.  The  models  were  used  to  evaluate 
transportation  alternatives,  including  variations  in  alternative 
transportation  modes  (air,  land,  sea)  and  sources  (military,  civilian, 
foreign),  as  well  as  factors  that  affect  transportation  mode  and  source 
decisions.  The  scope  of  the  MCS  described  the  study  as  an  assessment  of 
the  full  range  of  transportation  needs  required  to  support  (1)  combat 
operations;  (2)  smaller  military  efforts,  such  as  peacekeeping  or  overseas 
evacuation  of  American  citizens;  (3)  homeland  defense/civil  support,  such 
as  disaster  relief  and  antiterrorism  response;  and  (4)  other  strategic 
missions,  such  as  strategic  nuclear  and  global  strike  missions.  The  study 
was  coauthored  by  the  Office  of  the  Secretary  of  Defense,  Office  of  the 
Director,  Program  Analysis  and  Evaluation  (PA&E),  and  the  Chairman, 


4  The  2004  National  Military  Strategy  of  the  United  States  calls  for  a  force  sized  to  defend 
the  homeland,  proactively  dissuade  adversaries  in  and  from  four  global  regions,  and 
conduct  two  overlapping  “swift  defeat”  campaigns.  Even  when  committed  to  a  limited 
number  of  lesser  contingencies,  the  force  must  be  able  to  “win  decisively”  in  one  of  the  two 
campaigns.  This  “1-4-2-1”  force-sizing  construct  places  a  premium  on  increasingly 
innovative  and  efficient  methods  to  achieve  objectives. 
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Joint  Chiefs  of  Staff,  Office  of  the  Director  of  Logistics.  Other  DOD 
components  involved  in  the  study  included  the  U.S.  Transportation 
Command  and  its  subordinate  commands,  the  Office  of  the  Under 
Secretary  of  Defense  (Acquisition,  Technology  and  Logistics),  the 
combatant  commanders,  the  military  services,  and  others.  The  final  report 
was  signed  on  December  19,  2005,  by  the  Deputy  Secretary  of  Defense. 


Limitations  in  the 
MCS  Study  and 
Report  Raise 
Questions  about 
Adequacy  and 
Completeness 


As  measured  against  relevant  generally  accepted  research  standards, 
limitations  in  the  MCS  study  and  report  raise  questions  about  their 
adequacy  and  completeness.  For  example,  aspects  of  modeling  and  data 
were  inadequate  in  some  areas  because  data  were  lacking  and  some  of  the 
models  used  could  not  simulate  all  relevant  aspects  of  the  missions. 
Furthermore,  the  exclusive  use  of  warfighting  metrics  in  the  MCS  analyses 
limited  the  usefulness  of  the  report.  Moreover,  in  some  cases  the  MCS 
results  were  incomplete,  unclear,  or  contingent  on  further  study,  making  it 
difficult  to  identify  findings  and  evaluate  evidence.  Finally,  verification, 
validation,  and  accreditation  of  models  and  data  used  to  conduct  the  study 
were  incomplete  because  they  were  not  done  in  accordance  with  DOD 
policy  or  relevant  research  standards,  and  supporting  documentation  for 
key  processes  could  not  be  provided. 


Aspects  of  Modeling  and  Aspects  of  modeling  and  data  were  inadequate  in  some  areas  because  data 

Data  Were  Inadequate  were  lacking  and  some  of  the  models  used  could  not  simulate  all  relevant 

aspects  of  the  missions.  Relevant  research  standards  require  that  models 
used  are  adequate  for  the  intended  purpose,  represent  a  complete  range  of 
conditions,  and  that  data  used  are  properly  generated  and  complete.  As 
DOD  acknowledged  in  the  MCS  report  as  a  study  limitation,  some 
modeling  tools  were  not  available  to  analyze  key  missions.  The  MCS  cited 
deficiencies  in  several  existing  mobility  models  and  the  need  for  follow-on 
MCS  analyses.  The  MCS  report  also  acknowledged  that  the  identified 
deficiencies  in  data  hindered  analysis  of  future  requirements  and 
transportation  system  performance.  However,  the  report  did  not  explain 
how  these  limitations  could  affect  the  study  results  or  what  the  effect  on 
the  projected  mobility  capabilities  might  be. 

For  example,  the  MCS  modeled  hypothetical  homeland  defense  missions 
rather  than  homeland  defense  demands  derived  from  a  well  defined  and 
approved  concept  of  operations  for  homeland  defense,  because  the 
specific  details  of  the  missions  were  still  being  determined,  and  DOD 
acknowledged  that  the  data  used  are  incomplete.  The  MCS  report 
recommended  further  analysis  of  mobility  capabilities  after  homeland 
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defense  needs  are  refined.  However,  the  report  did  not  identify  the 
potential  effect  that  using  these  hypothetical  scenarios  might  have  on  the 
MCS  results.  The  MCS  also  was  unable  to  model  the  flexible  deterrent 
options/deployment  order  process  to  move  units  and  equipment  into 
theater  because  of  lack  of  data  on  how  deployment  orders  have  been 
issued  in  the  past  for  major  combat  operations.  However,  the  MCS 
assumed  a  robust  use  of  the  flexible  deterrent  option/deployment  order 
process,  which  in  one  scenario  accounted  for  approximately  60  percent  of 
the  early  airlift  movement  prior  to  the  beginning  of  combat  operations. 
Instead,  the  MCS  modeled  the  flow  of  forces  and  equipment  contained  in 
the  time-phased  force  deployment  data  process.5  Based  on  the  scenarios 
provided  for  the  MCS  analyses,  we  could  not  determine  how  the 
deployment  order  process  would  affect  the  mobility  assets  required  for 
major  combat  operations.  The  MCS  report  noted  that  additional  analysis  is 
required  to  determine  the  implications  of  the  deployment  order  process 
and  to  provide  sufficient  information  for  decision  makers  concerning  the 
amount  of  future  mobility  assets  actually  needed. 

In  addition  to  these  modeling  and  data  issues,  the  MCS  report  contains 
more  than  80  references  to  the  need  for  improved  modeling,  and  12  of 
these  references  call  for  additional  data  or  other  refinements,  such  as 
follow-on  analyses,  further  assessments,  future  evaluations,  additional 
study,  and  investigation  of  alternatives  in  a  wide  range  of  areas,  such  as 
antiterrorism  response,  infrastructure  availability,  intratheater  airlift, 
strategic  sealift,  air  refueling,  and  availability  of  civil  reserve  aircraft. 

Some  of  these  studies  are  currently  underway,  as  discussed  later  in  this 
report. 

Moreover,  our  analysis  of  the  MCS  report  showed  that  the  year  modeled 
(2012)  to  determine  the  DOD  transportation  capabilities  needed  for  the 
years  2007  through  2013  did  not  place  as  much  demand  for  mobility  assets 
in  support  of  smaller  military  operations,  such  as  peacekeeping,  as  other 
years.  To  establish  transportation  demands  for  mission  requirements,  DOD 
developed  and  used  a  baseline  security  posture6  that  covered  a  7-year 


5  In  joint  military  planning,  time-phased  force  deployment  data  are  defined  as  a  computer 
database  that  contains  detailed  personnel  and  cargo  planning  data;  it  usually  includes 
priority  and  sequencing  of  deploying  forces. 

6  The  baseline  security  posture  projects  the  position  from  which  combatant  commanders 
will  perform  future  missions,  including  how  they  will  address  the  global  war  on  terrorism, 
ongoing  operations,  and  other  day-to-day  activities  to  which  U.S.  forces  remain  committed 
and  from  which  they  are  not  likely  to  disengage  entirely. 
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period.  This  baseline  was  developed,  in  part,  using  a  historical  analysis  of 
DOD’s  movement  of  personnel,  equipment,  supplies,  and  other  items. 
According  to  DOD  officials,  Office  of  the  Under  Secretary  of  Defense  for 
Policy,  which  developed  the  baseline  security  posture,  selected  the  year 
modeled  in  the  MCS  because  it  was  deemed  the  “most  likely”  to  occur  in 
terms  of  transportation  demands  and  because  it  was  not  statistically 
different  from  other  years  in  the  7-year  period.  However,  our  analysis 
showed  that  2012  involved  the  least  demand  for  transportation  assets  in 
support  of  smaller  military  efforts  than  any  year  in  the  7-year  period  and 
did  not  fully  stress  the  defense  transportation  system.  Figure  1  depicts  the 
number  of  hypothetical  ongoing  contingencies  for  each  year  in  the 
baseline  as  shown  in  the  MCS. 


Figure  1:  MCS  Hypothetical  Ongoing  Contingencies  during  7-year  Baseline  Security 
Posture  Time  Frame 


Depiction  of  the  number  of  contingencies  from  2007  through  2013 


Year 


Modeled  year 


Source:  GAO’s  analysis  of  MCS  notional  lesser  contingency  schedule. 


Note:  A  particular  contingency  may  be  ongoing  in  more  than  1  year.  Each  contingency  has  unique 
cargo  and  passenger  requirements.  For  example,  a  contingency  that  may  be  ongoing  over  a  2-  or  3- 
year  time  frame  may  require  more  or  less  mobility  capability  than  a  6-month  contingency. 


Although  not  transparent  in  the  study,  DOD  officials  said  scenarios  in  the 
year  modeled  were  not  intended  to  fully  stress  the  defense  transportation 
system.  DOD  officials  provided  no  further  explanation  for  the  year 
selected  to  develop  the  DOD  transportation  capabilities  other  than  it  was 


Page  9 


GAO-06-938  Defense  Transportation 


directed  by  Office  of  the  Under  Secretary  of  Defense  for  Policy  and  agreed 
to  by  the  study  leadership.  We  believe  that  selection  of  a  different  year  that 
placed  an  increased  demand  on  transportation  assets  for  smaller  military 
efforts  may  have  revealed  gaps  in  mobility  requirements.  Therefore,  we 
found  that  the  selection  of  2012  as  the  modeling  year  was  a  limitation  in 
the  MCS  with  respect  to  smaller  military  efforts. 

Because  of  these  modeling  and  data  limitations,  the  MCS  may  have 
incorrectly  estimated  the  future  mobility  requirements  needed  to  support 
homeland  defense  missions,  major  combat  operations,  and  smaller 
contingencies.  Until  DOD  improves  aspects  of  the  modeling  and  data  used 
to  conduct  the  MCS — to  include  defining  its  homeland  defense  mission 
needs,  developing  models  for  the  deployment  order  process,  and 
explaining  how  identified  modeling  and  data  limitations  could  affect  the 
study  results — decision  makers  may  not  have  adequate  and  complete 
information  about  DOD’s  mobility  capabilities. 


Exclusive  Use  of 
Warfighting  Metrics  in 
MCS  Analyses  Limited 
Usefulness  of  Report 


While  the  MCS  concluded  that  combined  U.S.  and  host  nation 
transportation  assets  were  adequate  to  meet  U.S.  objectives  with 
acceptable  risk,  the  report,  in  describing  the  use  of  warfighting  metrics  in 
its  analyses,  does  not  provide  a  clear  understanding  of  the  direct 
relationship  of  warfighting  objectives  to  transportation  capabilities.  The 
report  acknowledged  that  further  analysis  is  required  to  understand  the 
operational  impact  of  increased  or  decreased  strategic  lift  on  achieving 
warfighting  objectives.  Relevant  generally  accepted  research  standards 
require  that  conclusions  be  supported  by  analyses.  The  use  of  warfighting 
metrics  is  a  measure  to  determine  whether  combat  tasks,  such  as 
establishing  air  superiority,  are  achieved.  However,  they  do  not  measure 
success  in  terms  of  whether  appropriate  personnel,  supplies,  and 
equipment  arrived  in  accordance  with  timelines.  As  a  result,  we  could  not 
determine  how  the  study  concluded  that  planned  transportation  assets 
were  adequate  because  the  study  did  not  contain  a  transparent  analysis  to 
support  its  conclusion.  In  our  opinion,  it  is  important  for  decision  makers 
to  have  an  understanding  of  both  combat  tasks  that  must  be  achieved  and 
the  amount  of  transportation  assets  needed  to  achieve  those  tasks  with 
some  level  of  success.  This  understanding  would  allow  creation  of  a  clear 
roadmap  for  investment  decisions.  However,  we  could  not  determine  how 
the  study  calculated  the  specific  numbers  of  transportation  assets  needed 
or  whether  there  are  specific  gaps,  overlaps,  or  excesses  in  transportation 
assets,  a  key  objective  of  the  study.  Previous  DOD  mobility  studies, 
including  the  Mobility  Requirements  Study — 2005,  primarily  used  mobility 
metrics,  which  measured  success  in  terms  of  tons  of  equipment  and 
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personnel  moved  per  day  to  accomplish  military  objectives.  Million-ton- 
miles  per  day  is  a  commonly  accepted  measure  of  airlift  performance  and 
reflects  how  much  cargo  can  be  delivered  over  a  given  distance  in  a  given 
period  of  time  based  on  the  capability  of  each  type  of  aircraft.  A  similar 
mobility  metric — short  tons — is  used  to  measure  ship  capability.  However, 
these  studies  did  not  fully  integrate  combat  tasks  as  a  metric.  The  use  of 
both  warfighting  and  mobility  metrics  to  measure  success  would  allow 
decision  makers  to  know  whether  there  is  sufficient  capability  to  achieve 
warfighting  objectives,  as  well  as  to  understand  the  number,  type,  and  mix 
of  mobility  assets  that  are  actually  needed. 


Results  Are  Not  Always 
Complete  or  Presented 
Clearly  and  Are  Qualified 
or  Contingent  on  Further 
Study  or  Analysis 


In  some  cases,  the  MCS  results  were  incomplete,  unclear,  or  contingent  on 
further  study,  making  it  difficult  to  identify  findings  and  evaluate  evidence. 
Relevant  research  standards  require  results  to  be  presented  in  a  complete, 
accurate,  and  relevant  manner;  conclusions  to  be  sound  and  complete;  and 
recommendations  to  be  supported  by  analyses.  Our  analysis  of  the  MCS 
report  found  that  it  contains  several  recommendations  for  further  studies 
and  assessments,  five  of  which  are  under  way.  The  five  studies  address 
intratheater  lift  capabilities;  sealift  petroleum,  oil,  and  lubricants;  logistics 
contingency  operations  capabilities;  aerial  refueling;  and  integrated  global 
presence  and  basing.  However,  the  report  does  not  explain  the  potential 
effect  of  these  ongoing  studies  on  the  MCS  results  after  the  studies  are 
complete,  nor  does  DOD  have  plans  to  report  the  effect  of  these  studies  on 
the  MCS  results. 


In  addition,  the  report  contains  qualified  information  that  is  not  presented 
clearly  in  the  report,  such  as  varying  assessments  of  intratheater  assets  in 
three  different  places.  For  example,  the  report  states  in  the  assessment 
section  of  the  executive  summary  that  projected  transportation  assets  are 
sufficient  to  address  intratheater  demands  in  the  fiscal  years  2007  through 
2013  time  frame.  However,  in  the  recommendations  section  of  the 
executive  summary,  the  report  states  that  DOD  should  take  action  to 
determine  the  proper  mix  of  intratheater  assets  needed  to  meet 
requirements.  Then,  in  the  part  of  the  report  that  discusses  intratheater 
airlift,  the  report  states  that  a  detailed  analysis  of  intratheater  airlift  needs 
would  require  improved  modeling  tools  to  accurately  capture  interactions 
among  land,  sealift,  and  airlift  capabilities  and  that  data  sets  must  be 
developed  that  accurately  describe  the  requirement  in  light  of  emerging 
concepts. 
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W&A  of  Models  and  Data 
Was  Not  Complete 


W&A  of  models  and  data  used  to  conduct  the  study  was  not  complete 
because  it  was  not  done  in  accordance  with  DOD  policy  or  relevant 
research  standards.  DOD  policy  issued  by  the  Under  Secretary  of  Defense 
for  Acquisition,  Technology  and  Logistics  requires  that  DOD  models  and 
data  go  through  a  W&A  process.  Moreover,  relevant  research  standards 
state  that  a  study  report  should  include  a  W&A  accreditation  report  that  is 
signed  by  the  study  director  and  addresses  the  models  and  data 
certification.  DOD  officials  acknowledged  that  they  did  not  comply  with 
the  W&A  policy  when  using  legacy  models  in  the  MCS  because  they 
believed  such  an  approach  was  not  warranted  for  legacy  models  that  have 
been  used  for  many  years  and  have  proved  reliable.  Moreover,  these 
officials  believe  that  such  long-term  use  constitutes  a  W&A  process 
equivalent  to  that  required  in  the  policy.  However,  the  DOD  policy  does 
not  specify  that  the  actual  use  of  a  model  constitutes  an  equivalent  W&A 
process.  W&A  of  models  and  data  reduces  the  risk  inherent  in  the  use  of 
models  and  simulations  by  improving  the  credibility  of  modeling  and 
simulation  results.  We  previously  reported  our  concerns  that  DOD  did  not 
follow  its  policy  in  executing  the  MCS  and  had  little  documentation  to 
support  the  W&A  process  used.7  We  found  that  the  final  MCS  report 
contained  a  description  of  the  equivalent  W&A  process,  but  DOD  officials 
could  provide  no  further  documentation  to  verify  and  validate  this 
equivalent  process  other  than  the  description  included  in  the  report.  We 
also  found  no  documentation  in  the  study  report  to  support  DOD  claims 
that  the  models  have  proven  reliable. 

Furthermore,  DOD  officials  were  unable  to  provide  documentation  to 
support  and  verify  key  analytical  and  decision-making  processes  used  by 
senior  DOD  leadership  throughout  the  study.  Relevant  research  standards 
support  documenting  the  study’s  analytical  and  decision-making  processes 
to  ensure  the  accuracy,  completeness,  and  credibility  of  study  results. 

DOD  officials  told  us  that  the  study’s  key  analytical  and  decision-making 
processes  were  validated  and  approved  by  study  participants  during 
working  group  meetings  and  by  senior  leadership  during  General  Officer 
Steering  Committee  meetings  and  Executive  Committee  meetings.  PA&E 
officials  could  not  produce  documentation  of  these  meetings  because  they 
said  documentation  did  not  exist.  Nor  could  they  produce  other 
documents  we  requested  during  the  development  of  the  MCS  or  following 


'  See  GAO,  Defense  Transportation:  Opportunities  Exist  to  Enhance  the  Credibility  of  the 
Current  and  Future  Mobility  Capabilities  Studies,  GAO-05-659R  (Washington,  D.C.:  Sept. 
14,  2005),  for  a  more  detailed  discussion. 
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Conclusions 


issuance  of  the  report.  Consequently,  we  were  unable  to  determine  the 
adequacy  and  completeness  of  the  analytical  and  decision-making 
processes  that  supported  the  MCS  effort  and  that  we  believe  are 
significant  to  the  credibility  of  the  study  and  its  conclusions. 


The  methodological  limitations  in  the  MCS  that  we  identified — some  of 
which  were  acknowledged  by  DOD  in  the  MCS  report — raise  questions 
about  the  adequacy  and  completeness  of  the  study  and  its  report.  Until 
DOD  improves  aspects  of  the  modeling  and  data  used  to  conduct  the 
MCS — such  as  defining  its  homeland  defense  mission  needs  and 
developing  models  for  the  deployment  order  process — decision  makers 
may  not  have  adequate  and  complete  information  about  DOD’s  mobility 
capabilities  to  enable  them  to  make  fully  informed  investment  decisions. 
Furthermore,  in  the  absence  of  an  explanation  of  how  identified  modeling 
and  data  limitations  could  affect  the  study  results  or  how  such  limitations 
could  affect  projected  mobility  capability  requirements,  the  accuracy  of 
the  study’s  finding  that  projected  capabilities  are  adequate  to  achieve  U.S. 
objectives  with  an  acceptable  level  of  risk  during  the  period  from  fiscal 
years  2007  through  2013  is  unclear.  Moreover,  without  a  transparent 
comparison  between  existing  mobility  assets  and  projected  needed  assets, 
decision  makers  will  be  unable  to  use  study  results  to  identify  and  quantify 
the  specific  types  and  mix  of  mobility  assets  needed  to  address  mobility 
gaps,  overlaps,  and  excesses.  Until  DOD  conducts  an  adequate  and 
complete  future  MCS  and  clearly  discloses  all  limitations  and  their  effects 
on  the  study  results,  decision  makers  likely  will  not  have  full  information 
concerning  DOD’s  mobility  capabilities.  As  a  result,  we  suggest  that 
Congress  and  other  decision  makers  exercise  caution  in  using  the  MCS  to 
make  programmatic  investment  decisions. 


Recommendation 
Executive  Action 


To  provide  decision  makers  with  adequate  and  complete  information 
concerning  mobility  capabilities  so  they  are  able  to  clearly  understand  the 
operational  implications  of  the  study  and  make  fully  informed 
programmatic  investment  decisions,  and  to  improve  the  usefulness  of 
future  mobility  capabilities  studies,  we  recommend  that  the  Secretary  of 
Defense  take  the  following  three  actions,  when  conducting  future  mobility 
capabilities  studies  beginning  with  any  study  currently  underway: 


•  develop  models  and  data  for  all  critical  missions,  such  as  homeland 
defense,  and  processes,  such  as  the  flexible  deterrent  options/deployment 
order  process; 

•  include  in  study  reports  an  explanation  of  how  stated  limitations  might 
impact  the  study  results  and,  at  a  minimum,  describe  how  recommended 
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future  studies  might  be  conducted  to  enhance  the  results  of  the  original 
study;  and 

•  incorporate  both  mobility  and  warfighting  metrics  in  determining 
capabilities. 


Agency  Comments 
and  Our  Evaluation 


In  commenting  on  a  draft  of  this  report,  DOD  concurred  with  our  first  and 
third  recommendations.  DOD  stated  it  did  not  understand  our  second 
recommendation  that  the  Secretary  of  Defense,  when  conducting  future 
mobility  studies,  beginning  with  any  study  currently  underway,  include  in 
study  reports  an  explanation  of  how  ongoing  and  follow-on  studies  and 
modeling  and  data  limitations  that  are  referenced  in  the  report  could  affect 
the  reported  results.  DOD  also  noted  that  it  plans  to  continue  its  ongoing 
efforts  to  enhance  the  models  and  data  collection  processes  used  to  assess 
mobility  capabilities  across  the  full  range  of  strategic  missions  and 
supports  the  notion  that  continual  improvements  are  needed.  As  we  noted 
throughout  our  report,  the  MCS  report  contains  numerous  and  repeated 
references  to  the  need  for  improved  modeling  and  additional  data  or  other 
refinements,  such  as  follow-on  analyses,  further  assessments,  future 
evaluations,  additional  study,  and  investigation  of  alternatives  in  a  wide 
range  of  areas.  DOD  further  commented  that  while  a  completed  study  can 
recommend  that  follow-on  studies  be  conducted,  it  cannot  explain  how 
future  studies  might  affect  the  results  of  the  current  study.  We 
acknowledge  that  DOD  cannot  quantitatively  predict  the  outcome  of  an 
ongoing  or  future  study.  However,  we  believe  DOD  should  be  able  to 
explain  what  ongoing  follow-on  studies  or  evaluations  seek  to  determine, 
what  changes  are  being  made  to  the  data  inputs  and  modeling  tools  that 
are  being  used  to  conduct  the  studies,  and  how  DOD  expects  the  results 
may  differ  from  current  study  results.  While  the  explanation  may  be 
hypothetical,  as  are  many  operations  research  study  hypotheses,  it  can 
provide  decision  makers  with  a  better  understanding  of  the  current  study’s 
limitations  and  results  and  how  an  ongoing  or  future  study’s  results  may 
differ.  Therefore,  we  refined  our  recommendation  to  recommend  that  the 
Secretary  of  Defense,  when  conducting  future  mobility  studies,  beginning 
with  any  study  currently  under  way,  include  in  study  reports  an 
explanation  of  how  stated  limitations  might  impact  the  study  results  and, 
at  a  minimum,  describe  how  recommended  future  studies  might  be 
conducted  to  enhance  the  results  of  the  original  study.  For  example,  if 
modeling  and  data  are  limitations  to  a  study,  the  report  should  discuss  the 
ways  in  which  the  results  might  change  with  better  models  and  data. 

DOD  also  commented  that  our  report  contained  misleading  information 
and  factual  errors  and  that  it  stands  by  the  adequacy  and  completeness  of 
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the  MCS.  The  department  provided  examples  in  its  technical  comments 
where  it  believed  our  report  contained  misleading  information  and  factual 
errors.  We  disagree  with  the  department’s  comments  regarding  the  facts 
in  our  report  and  have  addressed  each  of  the  department’s  comments  in 
appendix  II. 

Lastly,  DOD  stated  that  the  MCS  and  its  conclusions  are  well  accepted  by 
the  civilian  and  military  leadership  of  the  department,  and  pointed  out  that 
in  March  2006  testimony  before  the  House  Armed  Services  Committee,  the 
Commander,  U.S.  Transportation  Command,  stated  that  the  planned 
strategic  airlift  fleet  determined  by  the  MCS  is  “about  the  right  capacity”. 
However  we  note  that  in  the  same  hearing,  the  Commander  also  stated 
that  he  thought  DOD  needed  “somewhere  in  the  neighborhood  of’  20  C-17 
cargo  aircraft  beyond  what  is  planned.  We  also  note  that  in  the 
Commander’s  April  2006  testimony  before  the  Senate  Armed  Services 
Committee,  Subcommittee  on  Sealift,  he  stated  that,  in  an  internal 
Focused  Mobility  Analysis  to  study  strategic  mobility  from  a 
Transportation  Command  perspective,  the  MCS  will  be  the  baseline,  “but 
we  will  explore  how  changes  in  key  assumptions  may  impact  the 
analytical  outcome.” 


We  are  sending  copies  of  this  report  to  the  Secretary  of  Defense;  the 
Director  of  PA&E;  and  the  Office  of  the  Chairman,  Joint  Chiefs  of  Staff.  We 
will  also  make  copies  available  to  others  upon  request.  In  addition,  the 
report  is  available  at  no  charge  on  the  GAO  Web  site  at 
http://www.gao.gov.  If  you  or  your  staff  have  any  questions  regarding  the 
briefing  or  this  report,  please  contact  me  at  (202)  512-8366  or 
solisw@gao.gov.  Contact  points  for  our  Offices  of  Congressional  Relations 
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and  Public  Affairs  may  be  found  on  the  last  page  of  this  report.  GAO  staff 
who  made  key  contributions  to  this  report  are  listed  in  appendix  III. 


William  M.  Solis 

Director,  Defense  Capabilities  and  Management 
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List  of  Congressional  Committees 

The  Honorable  John  Warner 
Chairman 

The  Honorable  Carl  Levin 
Ranking  Minority  Member 
Committee  on  Armed  Services 
United  States  Senate 

The  Honorable  Ted  Stevens 
Chairman 

The  Honorable  Daniel  K.  Inouye 
Ranking  Minority  Member 
Subcommittee  on  Defense 
Committee  on  Appropriations 
United  States  Senate 

The  Honorable  Duncan  L.  Hunter 
Chairman 

The  Honorable  Ike  Skelton 
Ranking  Minority  Member 
Committee  on  Armed  Services 
House  of  Representatives 

The  Honorable  C.W.  Bill  Young 
Chairman 

The  Honorable  John  P.  Murtha 
Ranking  Minority  Member 
Subcommittee  on  Defense 
Committee  on  Appropriations 
House  of  Representatives 
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To  conduct  our  review  of  the  Mobility  Capabilities  Study  (MCS),  we 
reviewed  and  analyzed  the  final  MCS  report;  the  MCS  Terms  of  Reference; 
the  MCS  Study  Plan;  applicable  Department  of  Defense  (DOD)  strategic 
planning  guidance;  as  well  as  other  DOD  guidance,  directives,  instructions, 
and  memos  that  describe  how  DOD  would  conduct  its  MCS.  We  also 
reviewed  the  National  Security  Strategy  of  the  United  States  of  America 
and  the  National  Military  Strategy  of  the  United  States  of  America;  DOD 
guidance  concerning  data  collection,  development,  and  management  in 
support  of  strategic  analysis;  DOD  modeling  and  simulation  instruction; 
Defense  Modeling  and  Simulation  Office  guidance;  descriptions  of  models 
used  to  conduct  the  study;  and  the  databases  used  in  the  models.  We 
interviewed  study  officials  from  the  Office  of  the  Secretary  of  Defense, 
Program  Analysis  and  Evaluation  (PA&E),  and  the  office  of  the  Chairman, 
Joint  Chiefs  of  Staff,  Logistics,  as  well  as  study  participants  and  subject 
matter  experts  from  the  U.S.  Transportation  Command,  Air  Mobility 
Command,  Surface  Deployment  and  Distribution  Command,  the 
combatant  commands,  and  the  military  services  concerning  the  extent  of 
their  input  to  the  study.  We  also  interviewed  officials  from  the  Office  of 
the  Secretary  of  Defense,  Acquisition,  Technology  and  Logistics,  and  the 
Modeling  and  Simulation  Technical  Director  at  the  Defense  Modeling  and 
Simulation  Office. 

Additionally,  we  reviewed  research  literature  and  DOD  guidance  and 
identified  frequently  occurring,  generally  accepted  research  standards  that 
are  relevant  for  defense  studies  such  as  the  MCS  that  define  a  quality  or 
sound  and  complete  study.  The  following  were  our  sources  for  these 
standards: 

•  GAO,  Government  Auditing  Standards :  2003  Revision,  GAO-03-673G 
(Washington,  D.C.:  June  2003); 

•  GAO,  Designing  Evaluations,  GAO/PEMD-IO.1.4  (Washington,  D.C.: 

March  1991); 

•  GAO,  Dimensions  of  Quality,  GAO/QTM-94-1  (Washington,  D.C.:  February 
2004); 

•  RAND  Corporation,  RAND  Standards  for  High-Quality  Research  and 
Analysis  (Santa  Monica,  Calif.:  June  2004); 

•  Air  Force  Office  of  Aerospace  Studies,  Analysts  Handbook:  On 
Understanding  the  Nature  of  Analysis  (January  2000); 

•  Air  Force,  Office  of  Aerospace  Studies,  Air  Force  Analysis  Handbook,  A 
Guide  for  Performing  Analysis  Studies:  For  Analysis  of  Alternatives  or 
Functional  Solution  Analysis  (July  2004); 
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•  Department  of  Defense,  DOD  Modeling  and  Simulation  (M&S) 
Verification,  Validation,  Accreditation  (VV&A),  Instruction  5000.61 
(Washington,  D.C.:  May  2003); 

•  Department  of  Defense,  Data  Collection,  Development,  and  Management 
in  Support  of  Strategic  Analysis,  Directive  8260.1  (Washington,  D.C.: 
December  2,  2003);  and 

•  Department  of  Defense,  Implementation  of  Data  Collection,  Development, 
and  Management  for  Strategic  Analyses,  Instruction  8260.2  (Washington, 
D.C.:  January  21,  2003). 

During  the  process  of  identifying  generally  accepted  research  standards 
we  noted  that  not  all  studies  are  conducted  the  same  way.  For  example, 
while  all  studies  use  data,  not  all  use  baseline  data.  Likewise,  all  studies 
require  analyses,  but  not  all  use  models  or  simulation  to  conduct  analyses. 
We  tailored  the  research  standards  we  identified  as  relevant  to  the  MCS,  as 
shown  in  table  1. 


Table  1: 

Generally  Accepted  Research  Standards  Relevant  to  MCS  Requirements 

Design:  The  Study  is  well  designed 

1. 

Study  plan,  scope,  and  objectives  follow  Defense  Planning  Guidance 

I.a 

(Do  the  study  scope  and  objectives  fully  address  the  charter  presented  in  the  2004  Defense  Planning  Guidance?) 

l.a.1 

Does  the  study  plan  address  specified  guidance? 

l.b 

Was  the  study  plan  followed? 

l.c 

Were  deviations  from  the  study  plan  explained  and  documented? 

l.d 

Was  the  study  plan  updated  over  the  course  of  the  study  and  the  updates  explicitly  identified  in  the  study  and  updated 
study  plan? 

II 

Assumptions  and  constraints  are  reasonable  and  consistent 

II. a 

Are  assumptions  and  constraints  explicitly  identified? 

ll.a.1 

(Are  the  study  assumptions  necessary  and  reasonable?) 

II. b 

Do  the  study  assumptions  support  a  sound  analysis? 

II. c 

Are  the  assumptions  used  in  analyses  common  throughout  the  study  and  models? 

ltd 

Do  the  assumptions  contribute  to  an  objective  and  balanced  research  effort? 

III 

Scenarios  and  threats  are  reasonable 

III. a. 

Are  scenarios  traceable  back  to  formal  guidance? 

Ill.b 

Were  the  threat  scenarios  validated  and  Defense  Intelligence  Agency  approved  and  documented? 

lll.c 

Do  scenarios  represent  a  reasonably  complete  range  of  conditions? 

III  .d 

(Were  the  threats  varied  to  allow  for  the  conduct  of  sensitivity  analysis?) 

Execution:  The  study  is  well  executed 

IV 

Methodology  is  successfully  executed 

IV. a 

Was  the  study  methodology  executed  consistent  with  the  (MCS)  study  plan  and  schedule? 
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IV. b 

(Does  the  methodology  support  accomplishing  the  objectives  presented  in  the  study  plan?) 

IV. c 

Were  the  models  used  to  support  the  analyses  adequate  for  their  intended  purpose? 

IV. d 

Were  the  model  input  data  properly  generated  to  support  the  methodology? 

V 

(Analytical )  Baseline  data  and  other  data  used  to  support  study  and  analyses  validated,  verified,  and  approved 

V.a 

Is  the  (analytical)  baseline  fully  and  completely  identified  and  used  consistently  throughout  the  study  for  the  various 
analyses? 

V.b 

Were  data  limitations  identified  (and  the  impact  of  the  limitations  fully  explained?) 

V.c 

Were  the  (baseline  security  posture)  data  verified  and  validated? 

V.d 

Was  the  data  verification  and  validation  process  documented? 

VI 

Models,  simulations,  and  verification,  validation,  and  accreditation  are  reasonable 

VI. a 

Was  a  VV&A  accreditation  report  that  addresses  the  models  and  data  certification  signed  by  the  study  director  and 
included  in  the  report? 

VI.  b 

Were  modeling  and  simulation  limitations  identified  and  explained? 

VI. c 

Has  each  model  in  the  study  been  described? 

VI. d 

Are  the  model  processes  clearly  explained,  documented  and  understood? 

VII 

Measures  of  effectiveness  (MOEs)  and  essential  elements  of  analysis  (EEAs)  are  addressed 

VII. a 

(Do  MOEs  adhere  to  the  guidance  in  the  study  terms  of  reference?) 

VII. b 

(Are  the  MOEs  fully  addressed  in  the  study?) 

VII. c 

(Are  the  EEAs  addressed  in  the  study?) 

Presentation  of  results:  Timely,  complete,  accurate,  concise,  and  relevant  to  the  client  and  stakeholders 

VIII 

Presentation  of  results  support  findings 

VIII. a 

Does  the  report  address  the  objectives? 

VIII.  b 

Does  the  report  present  an  assessment  that  is  well  documented  and  conclusions  that  are  supported  by  the  analyses? 

VIII. c 

Are  conclusions  sound  and  complete? 

VIII. d 

Are  recommendations  supported  by  analyses? 

VIII. e 

Is  a  realistic  range  of  options  provided? 

Vlll.f 

Are  the  study  results  presented  in  the  report  in  a  clear  manner? 

VIII. g 

Are  study  participants/stakeholders  (i.e.,  services  and  Combatant  Commands)  informed  of  the  study  results  and 
recommendations? 

Source:  GAO  analysis  of  industry  and  DOD  study  and  research  standards. 


We  used  these  relevant  standards  as  our  criteria  to  assess  the  reported 
MCS  results.  All  eight  key  areas  of  the  study  process  were  considered  to 
have  equal  importance  relative  to  the  soundness  and  completeness  of  the 
study;  that  is,  a  sufficiently  serious  concern  in  any  category  could  raise 
questions  concerning  the  adequacy  and  completeness  of  the  report.  The 
analysts  independently  reviewed  evidence  relevant  to  each  subquestion, 
including  the  study  itself,  the  study  Terms  of  Reference,  and  its  strategic 
planning  guidance.  For  each  of  the  subquestions  in  the  key  study  process 
areas,  the  analysts  determined  whether  (1)  the  evidence  had  no  limitations 
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or  raised  no  concerns,  (2)  the  evidence  had  some  limitations  or  raised 
some  concerns,  (3)  the  evidence  had  significant  limitations  or  raised 
significant  concerns,  or  (4)  we  could  not  determine  the  extent  of 
limitations  or  concerns  because  there  was  not  sufficient  information.  The 
analysts  then  met,  compared,  and  discussed  their  individual  assessments, 
and  reached  an  overall  assessment  for  each  subquestion.  Areas  of  the 
study  where  we  identified  either  “some”  or  “significant”  limitations  or 
concerns  were  considered  to  affect  the  adequacy  or  completeness  of  the 
study.  Additionally,  areas  of  the  study  that  could  not  be  assessed  because 
of  the  lack  of  supporting  documentation  were  considered  to  affect  the 
credibility  of  the  study. 

Throughout  our  review  PA&E  officials  told  us  that  the  documentation 
needed  to  support  and  verify  the  key  analytical  and  decision-making 
processes  used  to  conduct  the  MCS,  documentation  that  was  vetted  and 
approved  by  DOD  leadership  and  all  of  the  study  participants,  would  not 
be  completed  and  available  for  our  review  until  the  study  report  was 
issued.  However,  after  the  report  was  issued,  we  were  told  that  the  report 
provides  all  of  the  supporting  documentation  needed  and  that  the  other 
documentation  we  requested  could  not  be  provided.  As  a  result,  we  were 
unable  to  determine  the  adequacy  and  completeness  of  the  analytical  and 
decision-making  processes  that  supported  the  MCS  effort  to  evaluate  the 
credibility  of  the  study.  We  believe  these  processes  are  significant  to  the 
credibility  of  the  study  and  its  results. 

We  conducted  our  review  from  July  2004  through  July  2006  in  accordance 
with  generally  accepted  government  auditing  standards. 
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DEPUTY  SECRETARY  OF  DEFENSE 

1010  DEFENSE  PENTAGON 
WASHINGTON,  DC  20301-1010 


AUG  3  0  2M6 


The  Honorable  David  M.  Walker 
Comptroller  General  of  the  United  States 
U.S.  Government  Accountability  Office 
441  G  Street,  N.W. 

20548 


This  is  the  Department  of  Defense  (DoD)  response  to  the  GAO  draft  report,  GAO- 
06-938,  “DEFENSE  TRANSPORTATION:  Study  Limitations  Raise  Questions  about  the 
Adequacy  and  Completeness  of  the  Mobility  Capabilities  Study  (MCS)  and  Report,”  dated 
August  21,  2006  (GAO  Code  350558).  DoD’s  responses  to  the  report’s  recommendations 
are  enclosed. 

The  GAO  draft  report  contains  misleading  information  and  factual  errors.  The 
Department  stands  by  the  adequacy  and  completeness  of  the  MCS,  which  is  one  of  the  most 
comprehensive  capability  studies  ever  conducted  by  the  Department  of  Defense.  The  data 
and  models  used  by  the  study  are  sound,  and  the  results  of  the  study  are  valid. 

Furthermore,  in  its  groundbreaking  assessment  of  homeland  defense  mission  needs,  the 
MCS  accurately  reflects  the  Department’s  most  current  understanding  of  this  developing 
national  mission.  Lastly,  MCS  insights  -  drawn  from  the  study’s  broad  array  of  mobility 
metrics  and  detailed  operational  simulations  -  significantly  enhanced  senior  leaders’ 
deliberations  during  the  2005  Quadrennial  Defense  Review. 

The  MCS  was  a  collaborative  effort  led  by  the  Office  of  the  Secretary  of  Defense  and 
the  Joint  Staff,  and  its  conclusions  are  well  accepted  by  the  civilian  and  military  leadership 
of  the  Department.  In  his  March  2006  Congressional  testimony,  the  Commander,  U.S. 
Transportation  Command  acknowledged  the  study’s  finding  by  stating  that  the  planned 
strategic  airlift  fleet  of  292  aircraft  “is  about  the  right  capacity.”  Likewise,  during  a  press 
conference  in  November  2005,  the  Vice  Chairman  of  the  Joint  Chiefs  of  Staff  expressed  his 
support  for  the  study,  citing  the  MCS  finding  that  the  Department  has  “a  very  capable  and 
adequate  airlift  fleet.” 


Enclosure: 
As  stated 
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UNCLASSIFED 

GAO  DRAFT  REPORT  -  DATED  August  21, 2006 
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DEPARTMENT  OF  DEFENSE  COMMENTS 
TO  THE  RECOMMENDATIONS 


RECOMMENDATION  1:  The  GAO  recommended  that  the  Secretary  of  Defense,  when 
conducting  future  mobility  capabilities  studies,  beginning  with  any  study  currently 
underway,  develop  models  and  data  for  all  critical  missions,  such  as  homeland  defense, 
and  processes,  such  as  the  flexible  deterrent  options/deployment  order  process. 

POD  RESPONSE:  DoD  concurs.  The  Department  plans  to  continue  its  ongoing  efforts 
to  enhance  the  models  and  data  collection  processes  used  to  assess  mobility  capabilities 
across  the  full  range  of  strategic  missions.  The  models  used  in  the  MCS  are  sound  and 
have  consistently  produced  valid  results  in  the  four  mobility  studies  conducted  by  DoD 
since  the  end  of  the  Cold  War.  Furthermore,  as  recommended  in  the  MCS  report,  DoD 
supports  the  notion  that  continual  improvements  are  needed  to  provide  enhanced  analytic 
tools  for  the  Department  as  it  strives  to  address  complex  real-world  processes. 

RECOMMENDATION  2:  The  GAO  recommended  that  the  Secretary  of  Defense,  when 
conducting  future  mobility  studies,  beginning  with  any  study  currently  underway,  include 
in  study  reports  an  explanation  of  how  ongoing  and  follow-on  studies  and  modeling  and 
data  limitations  that  are  referenced  in  the  report  could  affect  the  reported  results. 

DOD  RESPONSE:  DoD  does  not  understand  this  recommendation.  While  a  completed 
study  can  recommend  that  follow-on  studies  be  conducted,  it  cannot  explain  how  future 
studies  might  affect  the  results  of  the  current  study. 

RECOMMENDATION  3:  The  GAO  recommended  that  the  Secretary  of  Defense,  when 
conducting  future  mobility  capabilities  studies,  beginning  with  any  study  currently 
underway,  incorporate  both  mobility  and  warfighting  metrics  in  determining  capabilities. 

DOD  RESPONSE:  DoD  concurs.  In  fact,  the  MCS  employed  a  rigorous  process  by 
which  the  Services,  Combatant  Commands,  and  the  Joint  Staff  collaboratively  developed 
warfighting  metrics.  These  metrics  were  used  along  with  a  comprehensive  set  of 
mobility  metrics  for  airlift,  sealift,  surface  transport,  and  prepositioned  equipment  to 
determine  the  adequacy  of  the  full  spectrum  of  mobility  capabilities. 
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See  comment  1 . 


See  comment  2. 


See  comment  3. 
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DEPARTMENT  OF  DEFENSE  TECHNICAL  COMMENTS 

The  following  are  three  examples  of  factual  errors  and  misleading  information  contained  in  the 
draft  GAO  report: 

1.  Page  3:  “The  [modeled]  year  of  2012  did  not  place  as  much  demand  for  mobility  assets  in 
support  of  smaller  military  operations,  such  as  peacekeeping,  as  other  years.” 

Response:  False.  The  seven-year  demand  (2007-2013)  developed  as  part  of  the  Baseline  Security 
Posture,  and  used  by  the  MCS,  does  not  have  significant  variance  from  year  to  year.  2012  demand  is 
not  significantly  less  than  2009,  and  is  larger  than  2013. 

It  is  important  to  note  the  fact  that  the  MCS  modeled  a  surge  demand  on  the  mobility  system  in 
2012  that  far  exceeds  anything  this  nation  has  experienced  since  World  War  II,  In  doing  so,  the  study 
correctly  applied  the  Department’s  strategic  planning  framework  (1-4-2-1)  and  its  guidance  with 
respect  to  the  conduct  of  concurrent  lesser  contingencies  during  overlapping  war  fights.  GAO  has 
incorrectly  focused  on  the  number  of  operations,  not  the  level  of  effort. 

2.  Page  4:  “The  use  of  war  fighting  metrics  is  a  measure  to  determine  whether  combat  tasks, 
such  as  achieving  air  superiority,  are  achieved.  However,  they  do  not  measure  whether 
appropriate  personnel,  supplies,  and  equipment  arrived  in  accordance  with  timelines.” 

Response:  False.  The  war  fighting  metrics  developed  by  the  MCS  do  measure  whether 
appropriate  personnel,  supplies,  and  equipment  arrived  in  accordance  with  timelines. 

In  fact,  the  MCS  employed  a  rigorous  process  to  develop  war  fighting  metrics  that  were  used 
along  with  a  comprehensive  set  of  mobility  metrics  for  airlift,  sealift,  and  surface  transport.  These 
metrics  were  used  to  determine  if  the  modeled  war  fights  accomplished  the  commander’s  objectives 
within  the  right  timelines.  As  we  explained  to  GAO,  being  able  to  achieve  a  desired  task  within  the 
desired  timeline  requires  the  appropriate  personnel,  supplies  and  equipment  to  be  in  place  on  time, 
indicating  that  the  transportation  capabilities  are  adequate. 

3.  Page  3:  “The  MCS  also  was  unable  to  model  the  flexible  deterrent  options/deployment 
order  process  . . .” 

Response:  False.  The  MCS  DID  analyze  flexible  deterrent  option  (FDO)  movements  to  the 
theater.  As  part  of  the  MCS  analysis,  the  study  modeled  the  flow  of  forces  in  response  to  rising 
tensions  in  various  regions  of  the  world.  Forces  were  flowed  to  the  region  in  anticipation  of  operations 
before  war  plans  were  executed.  These  flexible  deterrent  options  were  included  in  the  MCS  analysis. 
What  the  study  did  not  do  was  model  the  deployment  order  process  used  in  OIF.  Rather,  the  study 
used  the  time-phased  force  deployment  data  (TPFDD)  process  as  the  accepted  methodology  for 
flowing  follow-on  forces. 
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4.  Page  3:  “The  MCS  modeled  hypothetical  homeland  defense  missions  rather  than  actual 
SG6  comment  4.  homeland  defense  demands  because  the  specific  details  of  the  missions  were  still  being 

determined,  and  DoD  acknowledged  that  the  data  used  may  be  incomplete.” 

Response:  The  statement  is  misleading.  We  are  not  sure  what  the  report  means  by  “actual 
homeland  defense  demands.”  Actual  demands  are  encountered  in  response  to  actual  events.  The  MCS 
was  intended  to  inform  the  DoD  leadership  concerning  the  impact  of  potential  demand  on  the  mobility 
system,  and  to  assess  the  risks  associated  with  different  potential  demand  levels.  The  study  used  the 
latest  approved  homeland  defense  scenarios  developed  by  NORTHCOM  and  PACOM,  as  well  as  the 
Department’s  current  planning  guidance,  to  determine  the  range  of  mobility  assets  needed  to  support  a 
range  of  missions.  Given  the  many  unknowns  associated  with  homeland  defense,  the  study  assessed 
low,  moderate,  and  high  levels  of  DoD  support  for  these  missions. 


See  comment  5.  5.  Page  3:  “Aspects  of  modeling  and  data  were  inadequate . . 

Response:  This  statement  is  misleading.  These  are  the  same  models  that  the  Department  has 
relied  on  to  complete  three  previous  mobility  studies  since  the  end  of  the  Cold  War.  Furthermore,  the 
Department  has  dedicated  significant  resources  to  make  sure  that  these  models  and  the  data  collection 
processes  are  adequate.  The  fact  that  the  study  recommends  future  improvements  was  meant  to  focus 
future  enhancements  as  the  Department  strives  to  address  increasingly  complex  real  world  issues.  The 
MCS  models  and  data  were  adequate  to  assess  relevant  aspects  of  the  missions  required  to  support  the 
National  Military  Strategy, 
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GAO’s  Responses  to 
DOD’s  Technical 
Comments 


1.  DOD  disagreed  with  our  assessment  that  the  modeled  year — 2012 — did 
not  place  as  much  demand  for  mobility  assets  in  support  of  smaller 
military  operations,  such  as  peacekeeping,  as  other  years.  DOD  also 
stated  that  we  incorrectly  focused  on  the  number  of  operations,  not 
the  level  of  effort.  We  disagree.  The  MCS  report  (Annex  A  to  Appendix 
F)  made  no  distinction  between  the  number  of  lesser  contingencies 
and  the  level  of  effort.  Specifically,  the  Vignettes  for  Baseline  Security 
Posture  Analysis  did  not  report  the  level  of  effort  by  year  and  instead 
aggregated  the  data,  in  many  instances  across  several  modeled  years. 
Consequently,  we  compared  the  number  of  operations  conducted  in 
the  model  year.  Throughout  our  review,  PA&E  officials  consistently 
told  us  that  the  completed  MCS  report  would  contain  all  the 
documentation  needed  to  support  its  analyses.  Furthermore,  although 
demand  in  the  modeled  year  may  exceed  previous  efforts,  the  MCS 
was  chartered  to  assess  the  ability  of  the  mobility  system  to  support 
the  National  Military  Strategy  into  the  next  decade.  The  size  of  the 
selected  model  year  in  relation  to  efforts  conducted  between  1941  and 
2006  is  not  at  issue.  As  our  report  makes  clear,  our  concern  is  that 
modeling  what  appears  to  be  the  least  demanding  year  does  not 
address  whether  the  United  States  has  sufficient  capability  to  support 
national  objectives  during  a  peak  demand  period  and  may 
underestimate  and  underreport  demands  to  senior  decision  makers. 

2.  DOD  disagreed  with  our  observation  that  the  MCS  report  does  not 
provide  a  clear  understanding  of  the  direct  relationship  of  warfighting 
objectives  to  transportation  capabilities.  We  disagree.  We  understand 
that  achieving  a  combat  task  requires  delivering  the  right  commodity 
to  the  right  place  at  the  right  time.  However,  the  specific  combat  tasks 
(e.g.,  attaining  air  superiority)  necessary  to  satisfy  the  commander’s 
campaign  objectives  are  not  a  direct  measure  of  mobility  capability. 
For  example,  the  problems  in  using  a  single  metric  are  reflected  in  the 
MCS  Appendix  H,  where  the  MCS  report  states  that  “the  study  itself 
still  had  difficulty  in  evaluating  the  operational  impact  of  the  delivery 
of  theater  support  elements,”  adding  that  “we  [DOD]  were  unable  to 
develop  a  satisfactory  mechanism  to  capture  the  linkage  of  the  closely 
related,  but  delayed,  follow-on  support  needed...”  Finally,  the  MCS 
concludes  that  “there  was  no  way  to  model  a  decrease  in  [Air  Force] 
squadron  effectiveness  if  this  support  was  late.  Additional  effort  is 
required  to  develop  a  methodology  for  evaluation  the  operational 
impact  of  support  equipment  availability.”  We  continue  to  believe,  and 
DOD  agreed  with  our  recommendation,  that  warfighting  metrics,  in 
conjunction  with  mobility  metrics,  can  give  decision  makers  a  full 
picture  of  the  capabilities  needed  to  meet  a  specific  warfighting  goal. 
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3.  DOD  disagreed  with  our  statement  that  the  MCS  was  unable  to  model 
the  flexible  deterrent  options/deployment  order  process  and  that  the 
study  in  fact  analyzed  flexible  deterrent  option  (FDO)  movements  to 
theater.  We  do  not  dispute  that  DOD  analyzed  FDO  movements  as  part 
of  the  MCS  analysis.  However,  the  degree  to  which  the  MCS  analyses 
successfully  modeled  FDOs  is  in  question.  The  MCS  report,  Appendix 
H,  stated  that  an  individual  FDO  is  essentially  the  same  as  a 
deployment  order.  It  also  states  in  that  section  that  “Deployment 
orders  [DEPORDS]  are  issued  to  deploy  specific  capabilities  as 
commitment  decisions  are  made.  This  was  not  modeled  due  to  lack  of 
data  on  how  DEPORDS  would  have  been  issued  for  an  MCO  [major 
combat  operation]  deployment.”  In  the  same  paragraph,  the  MCS 
concludes  that  “the  impact  on  the  mobility  system  of  the  DEPORD 
process  should  be  assessed  in  follow-on  MCS  analyses,”  adding  that 
“there  is  a  data  deficit  on  how  to  model  and  execute  a  DEPORD 
process.”  Furthermore,  the  MCS  report  states  that  “additional  analysis 
is  required  to  investigate  the  implications  of  the  DEPORD  process 
decisions  and  provide  data  for  future  decision-makers  to  develop  a 
DEPORD  execution  process.” 

4.  DOD  believes  our  statement  concerning  homeland  defense  missions  is 
misleading  and  is  not  sure  what  the  report  means  by  “actual  homeland 
defense  demands.”  We  removed  the  word  “actual”  and  clarified  our 
report  to  discuss  “demands  derived  from  a  well  defined  and  approved 
concept  of  operations  for  homeland  defense”,  which  were  not 
available  for  the  study  according  to  the  MCS  report.  Furthermore,  in 
chapter  4,  the  MCS  report  states  that  “maintaining  a  dedicated 
capability  to  support  multiple,  nearly  simultaneous  homeland 
defense/civil  support  events  concurrent  with  the  peak  demand  period 
of  two  overlapping  warfights,  greatly  exceeds  programmed  lift 
capabilities”.  This  raises  questions  about  the  conclusions  of  the  MCS 
that  there  are  adequate  mobility  capabilities  to  meet  national  security 
objectives.  Also,  in  Chapter  3  of  the  MCS  report,  it  states  that  the  DOD 
homeland  defense  concept  of  operations  required  refinement  and  was 
one  of  nine  issue  areas  cited  within  the  homeland  defense  portion  of 
the  study  that  “need  to  be  addressed  and  investigated  in  more  detail.” 
All  of  these  nine  areas  potentially  impact  mobility  support  for 
homeland  defense  operations.  Moreover,  the  MCS  Executive  Summary 
notes  that  reassessment  of  these  missions  is  required  as  DOD’s  role  in 
homeland  defense  evolves.  The  MCS  report,  chapter  4,  concludes  by 
calling  for  further  refinement  of  mission  requirements,  continuing  risk 
assessments,  and  an  effort  to  determine  corresponding  mobility 
solutions.  We  continue  to  believe  that  the  MCS  conclusion  that 
adequate  mobility  capability  exists  is  conditional  given  the  results  of 
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the  homeland  defense  portion  of  the  study  and  that  the  accuracy  and 
completeness  of  the  data,  modeling,  and  results  for  this  portion  of  the 
MCS  remain  in  question. 

5.  We  disagree  with  DOD’s  characterization  that  our  information  was 
“misleading”  regarding  the  adequacy  of  some  aspects  of  the  MCS’ 
modeling  and  data.  Furthermore,  we  continue  to  disagree  with  DOD’s 
statement  that  the  models  and  data  used  by  the  study  were  sound  and 
adequate  to  assess  relevant  aspects  of  missions  required  to  support  the 
National  Military  Strategy,  and  that  the  results  of  the  study  are  valid.  In 
this  report,  as  in  our  September  14,  2005  report,1  we  reaffirm  our 
concern  that  the  data  and  models  used  by  the  study  may  not  be  sound 
and  the  results  may  not  be  valid  since  the  verification,  validation,  and 
accreditation  (W&A)  of  the  models  and  data  used  to  conduct  the 
study  was  not  done  in  accordance  with  DOD  policy  or  relevant 
research  standards.  W&A  of  models  and  data  reduces  the  risk 
inherent  in  the  use  of  models  and  simulations  by  improving  the 
credibility  of  modeling  and  simulation  results.  We  do  not  dispute 
DOD’s  assertion  that  it  has  relied  upon  the  same  models  to  produce 
mobility  studies  done  “since  the  end  of  the  Cold  War”.  However,  as  we 
discuss  in  our  report,  the  MCS  report  fails  to  explain  or  qualify  the 
impact  that  identified  data  or  modeling  limitations  might  have  on  its 
results.  For  example,  in  the  MCS  chapter  4,  entitled  Operational  Data, 
the  MCS  states  that  “data  deficiencies  negatively  affected  MCS’s  ability 
to  use  current  execution  data  to  project  future  requirements  and 
assess  system  performance.”  Unclear  is  the  extent  to  which  these 
deficiencies  impacted  the  MCS  ability  to  meet  the  objective  of 
identifying  mobility  capability  gaps,  overlaps,  or  excesses  and  provide 
associated  risk  assessments,  an  MCS  objective.  Similarly,  in  the 
section  of  chapter  4  entitled  Analysis  Tools,  the  report  states  that 
“MCS  analysis  revealed  several  deficiencies  in  existing  mobility 
models.”  The  section  concludes  with  five  recommended  tool 
enhancements  but  it  does  not  explain  the  impact  that  the  absence  of 
these  enhanced  tools  may  have.  We  continue  to  believe  that  because  of 
these  modeling  and  data  limitations,  the  MCS  may  have  incorrectly 
estimated  the  future  mobility  requirements  needed  to  support 
homeland  defense  missions,  smaller  contingencies,  and  major  combat 
operations. 


1  See  GAO,  Defense  Transportation:  Opportunities  Exist  to  Enhance  the  Credibility  of  the 
Current  and  Future  Mobility  Capabilities  Studies,  GAO-05-659R  (Washington,  D.C., 
September  14,  2005),  for  a  more  detailed  discussion. 
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